From Competence to Efficiency: A Tale 
of GA Progress 


David E. Goldberg 
Department of General Engineering 
University of Illinois at Urbana-Champaign 
Urbana, IL 




FROM COMPETENCE TO EFFICIENCY: 
A TALE OF GA PROGRESS* 


David E. Goldberg 
Department of General Engineering 
University of Illinois at Urbana-Champaign 
Urbana, IL 61801 
e-mail: deg@uiuc.edu 


INTRODUCTION 

Genetic algorithms (GAs) - search procedures based on the mechanics of natural selection and 
genetics - have grown in popularity for the solution of difficult optimization problems. Concomitant 
with this growth has been a rising cacaphony of complaint asserting that too much time must be spent 
by the GA practitioner diddling with codes, operators, and GA parameters; and even then these GA 
cassandras continue, and the user is still unsure that the effort will meet with success. At the same 
time, there has been a rising interest in GA theory by a growing community - a theorocracy - of 
mathematicians and theoretical computer scientists, and these individuals have turned their efforts 
increasingly toward elegant abstract theorems and proofs that seem to the practitioner to offer little in 
the way of answers for GA design or practice. 

What both groups seem to have missed is the largely unheralded 1993 assembly of integrated, 
applicable theory and its experimental confirmation. This theory has done two key things. First, it has 
predicted that simple GAs are severely limited in the difficulty of problems they can solve, and these 
limitations have been confirmed experimentally. Second, it has shown the path to circumventing these 
limitations in nontraditional GA designs such as the fast messy GA. 

This talk surveys the history, methodology, and accomplishment of the 1993 applicable theory 
revolution. After arguing that these accomplishments open the door to universal GA competence , the 
paper shifts the discussion to the possibility of universal GA efficiency in the utilization of time and real 
estate through effective parallelization, temporal decomposition, hybridization, and relaxed function 
evaluation. The presentation concludes by suggesting that these research directions are quickly taking 
us to a golden age of adaptation. 


*Effort sponsored by the Air Force Office of Scientific Research, Air Force Material Command, USAF, under grant 
number F49620-94-1-0103 and grant number F49620-95- 1-0338. The views and conclusions contained herein are those 
of the author and should not be interpreted as necessarily representing the official policies or endorsements, either 
expressed or implied, of the Air Force Office of Scientific Research or the U.S. Government. 
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MOTIVATION 


Complaint: GAs are too heuristic, too hit and 
miss. 

Practitioners diddle with codes, ops, and 
parameters. 

Theoreticians diddle with theorems, proofs, and 
Markov chains. 

Have GAs to solve hard problems quickly and 
reliably. 

Have applicable theory to answer questions of 
practice. 

Need to solve mega-problems with mega- 
machines. 



OVERVIEW 


A brief history lesson. 

A brief lesson of methodology. 
How the West was won. 

How efficiency will be won. 



WHAT ARE GAs? 


A genetic algorithm is a search procedure based on 
the mechanics of natural selection and natural 
genetics. 

To be a GA in sense of Holland (1975), two things 
are required: 

1) selection 

2) recombination 



SELECTION 


Make more copies of better guys. 

Lots of ways to do: roulette wheel, ranking, 
tournaments, etc. 


With niching or without. 



PROPORTIONATE REPRODUCTION 


Redistribute population according to /. 



Like spinning a weighted roulette wheel. 



Other ways: tournaments, ranking, expected 
number. 


RECOMBINATION 


Combine bits and pieces of solutions to form 
new, possibly better, solutions. 


Lots of ways to do: string-wise, tree-wise, single 
or multiple points. 


Example, single-point crossover: 



WHY DO GAs WORK? 


Intuitive version. 

GA Power = Reproduction + Recombination. 
It is something like human innovation. 
Combining notions to form ideas. 


Can we make this more rigorous? 



A (TOO) SHORT GA HISTORY 


The Zeitgeist of cybernetics. 
Holland's swashbuckling vision. 
1975: A very good year. 

1993: The hits just keep on coming. 



THE GHOST OF CYBERNETICS PAST 


The 1 950s and 1 960s were exciting times. 
Digital computation comes alive. 

EN I AC (1945). 

Perceptrons and other neural nets. 


Various dreams of digital evolution. 



HOLLAND S SWASHBUCKLING VISION 


Iterative circuit computers (1958, 1959). 

Pathbuilding addressing scheme. 

Problem: What to do with these computers? 

Create bands of roving programs. 

Fighting, mating, loving, dying, and competing 
for real estate and time. 


Emphasis on exchange. 



OTHER CONTEMPORARY VISIONS 


Rechenberg: Hardware evolutionsstrategie. 

Schwefel: Software evolutionstrategie. 

Fogel, Owens and Walsh: Dueling FSMs. 

Emphasis on genetic hill climbing: selection 
mutation. 



CHATEAU HOLLAND, 1975 


Adaptation in Natural and Artificial Systems, 
1975. 

Building blocks are the key. 

Bits and pieces of solutions recombined to 
speculate on better solutions. 

Facet wise mathematical theory. 


How the theory applies to economics, 
optimization, machine learning and other 
complex systems. 



ANOTHER GOOD YEAR: 1993 


Completion of GA design decomposition 
(Goldberg, 1991) 


Mixing results (Goldberg, Deb and Thierens, 
1993; Thierens and Goldberg, 1993): Simple 
GAs mixing limited. 


fm GA results (Goldberg, Deb, Kargupta and 
Harik, 1993): Nontraditional GAs can solve hard 
problems quickly and reliably. 



ERAS 


The Era of Visions (1950-something to 1975) 

The Era of Knowledge (1975 to 1985) 

The Era of Competence (1985 to 1993) 

The Era of Efficiency (1 993 to ???) 

The Golden Age of Automated Innovation (20xx 
to ???). 



HOW WE GOT HERE: SOME THOUGHTS 
ON METHODOLOGY 


Invention and engineering, not science. 

Require economy of modeling, not grand 
unifying theories. 


Example: The Wright Brothers 



MATERIAL VS. CONCEPTUAL MACHINES 


O'Hare story. 

Why change the ground rules - material to 
conceptual. 

Design is design is design. 

Philosophy of engineering as antidote to 
philosophy of science. 


Economy of modeling critical. 



LESSON OF THE WRIGHT BROTHERS 


Effective design decomposition of your problem. 

Facet wise, economic models of subproblem 
facets. 

Bounding empirical study and calibration. 
Scaling laws (dimensional analysis) important. 



A CARICATURE OF COMPETENCE 


Design decomposition. 

Scaling laws of time and space. 
Results: 

1) limits of sGAs 

2) efficiency of linkage-friendly GAs. 



GA DESIGN DECOMPOSITION 


Know what GA processes: building blocks 
(BBs). 

Ensure BB supply. 

Ensure BB growth and speed. 

Ensure good BB decisions. 

Ensure good BB mixing (exchange). 

Know BB challengers. 



WHAT ARE WE PROCESSING? 


Similarities among strings. 

Schemata are similarity subsets. 

Schemata described by similarity templates. 

Example: *1*** = {strings with 1 in second 
position}. 

Population contains schemata. 


String 

Fitness 

10111 

10 

01000 

5 

11010 

3 

00011 

20 




TEMPORAL YARDSTICK 


Analysis of selection time alone. 
Must-innovate time. 

Compare to other times. 





SPATIAL YARDSTICKS 


Population sizing. 

«o = c(a) p 2 mx 2 ->n 0 = O(^) 

Good decisions vs. stepwise decisions. 
Innovation vs. hill climbing. 




Confidence factor, £ 

5 G 91 144 214 328 540 1084 

Population size, n 

Figure 1 - Simulation results for FI with l = 200, presented on a 
graph of convergence as measured by the average number of 
correct alleles versus confidence (population size). The results 
are consistent with the i = 20 and 1 = 50 simulations, and the SUS 
and ranking results show more pronounced margins above the 
expected lower bound than the runs at lower t values. 


271 




THE COMPLEXITY TEMPTATION 


• Serial work, W , is proportional to nt 

• n = 0(f) 

• t = 0(log ft) 

• W = o(/t log ft) 
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Function evaluations 



Figure 2 - The total number of function evaluations for each 
selection scheme, graphed versus l value on log-log axes at C, 
= 0.9 for function FI . The total number of function evaluations 
varies approximately as ^ 1 - 7 in the pushy (ranking and 
tournament) selection schemes and l 2 - 3 in the purely 
proportionate (SUS and roulette) schemes. 




Ideal failure boundaries 



Selection pressure, s 


Figure 3 - A control map of a GA shows the drift, mixing, and 
cross-competitive limits of GA success. Within those 
boundaries, a GA should be expected to work well. 


Sinulation results 
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Figure 4 - Simulation results of simple GAs. An average of 50 
runs is plotted. Points for s values smaller than 1 .2 are 
obtained with a population size 21 (70% confidence) and other 
points are obtained with a population size 160 (99% 
confidence). 
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Messy GA Complexity 



50 


70 100 

Problem length 




AN EFFICIENCY SKETCH 


Real estate 

Time 

Sampling 

Hybrids and evaluation efficiencies 



REAL ESTATE: EFFICIENT PARALLEL GAs 


Studies hung up on topology or migration rate. 

Sizing in primary issue (Goldberg, Kargupta, 
Horn and Cantu, 1995) 

Two models: complete isolation and perfect 
mixing. 

Two constraints: fixed reliability and fixed 
computation. 



TIME: CONVERGING NO BB BEFORE ITS TIME 


BBs converge reliably in favorable signal-to- 
noise. 

Can we sustain diversity or inject lost BBs at right 
time? 

Mutation-like ops, dominance and diploidy, and 
niching. 


Value of mutation here. 



SAMPLING 


Discrete event simulation: Accurate-costly 
evaluation vs quick-dirty. 

How to trade off? 

Past studies: quick and dirty wins. 

Miller and Goldberg 1995. Theoretical 
investigation quantifies. 



EFFICIENT HYBRIDS 


All real applications are hybrids. 

How do we combine different techniques 
efficiently? 

Need economic balance of work. 

Likewise need efficient parallelization of 
evaluation. 



SUMMARY 


History, methodology, competence and 
efficiency. 

Fast, reliable GAs are here. 

Applicable theory is here. 

Guidelines for efficiency are on the way. 



CONCLUSIONS 


Practitioners must become aware of these 
results 

Or face a life of diddling. 

Theoreticians must understand this methodology 

Or be doomed to staring at Markov chains. 

Embrace will take us to a golden age of 
automated innovation. 



